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(57) ABSTRACT 

A communications network test system facilitates autono- 
mous or attendant-free interaction between the administra- 
tive interfaces of multiple network devices under test. The 
test system includes device-specific communication inter- 
face packages that map generic commands to device-specific 
commands. A generic package includes generic procedures 
that access the device-specific packages to perform common 
functions, such as startup and cleanup. Test cases can thus be 
written using the generic commands without requiring the 
tester to have knowledge of device-specific demands. In 
addition, multiple devices can be simultaneously tested and 
monitored using a single test platform. 
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METHODS AND SYSTEMS FOR TESTING 
COMMUNICATIONS NETWORK COMPONENTS 

RELATED APPLICATIONS 

[0001] This application claims the benefit of U.S. Provi- 
sional Patent Application No. 60/243,691 filed Oct. 27, 
2000, the disclosure of which is incorporated herein by 
reference in its entirety. 

TECHNICAL FIELD 

[0002] The present invention relates to the testing of a 
communications network. More particularly, the present 
invention relates to methods and systems for automated, 
integrated testing of multiple devices in a communications 
network. 

BACKGROUND ART 

[0003] Wireless and wireline communications networks 
have evolved to encompass a variety of services, switching 
platforms, applications, data processing system hardware 
and equipment that are collectively referred to as network 
products. Before deploying such network products within a 
live network environment, it is important to first test these 
network products to ensure that their operation is as 
expected. Typical telecommunications environments rely 
upon network elements (e.g., packet switches, database 
nodes, routers, etc.) produced by a variety of manufacturers, 
and such network elements may each further include a 
number of operating system or application software revi- 
sions. Consequently, one of the primary objectives or goals 
of the testing process prior to deployment is to identify and 
resolve any potential problems in the provisioning of net- 
work services that may result from the diverse nature of the 
collection of network elements. Such problems may include, 
for example, compatibility problems between newly devel- 
oped network elements and existing or legacy equipment/ 
software. 

[0004] A complete test regimen may involve multiple test 
platforms that are required to perform several tests in 
parallel on several switches, with a variety of resources 
being required to perform each test. For example, test 
platforms may require connections to several switches in 
order to send commands corresponding to a test, and to 
subsequently collect results of the test. From an operational 
standpoint, providing a dedicated connection from each test 
platform to each switch is typically not practical due to the 
large number of dedicated connections that would be 
involved. Furthermore, a particular test platform may not be 
capable of being physically connected to multiple switches 
at the same time. 

[0005] As an alternative to simultaneous permanent con- 
nections, a temporary connection may be manually estab- 
lished from a test platform to a device under test (DUT) 
when the test platform needs to communicate with a test 
program executing on the DUT. However, such manual 
connection techniques are time consuming and inefficient, 
and consequently are often deemed unacceptable by network 
operators. 

[0006] With particular regard to test platforms, it will be 
appreciated that such equipment can be driven by a data 
processing system software application that communicates 



with the DUT via commands over a serial connection. A 
problem with such test devices is that they typically require 
a command line interface dedicated to accomplishing a 
narrowly defined test objective and is therefore both cum- 
bersome and time consuming to use. Scripted batch files that 
automate a series of commands from a test platform to the 
DUT can be used to alleviate this problem, as the use of 
batch files does not require manual entry of commands to the 
connected test device. Consequently, a single batch file, or 
small set of batch files, which contains all the commands 
necessary to accomplish a particular test objective may be 
employed to execute a complicated sequence of test instruc- 
tions, thereby freeing the test operator to perform other tasks 
for duration of the test sequence. However, batch files must 
still be written using device-specific instructions and require 
that the operator have detailed knowledge of the command 
line interface commands of each device being tested. 

[0007] FIG. 1 is a schematic diagram illustrating a con- 
ventional architecture associated with the testing of a tele- 
communications packet switch. FIG. 1 includes a test archi- 
tecture, generally indicated by reference numeral 100, that is 
comprised of a test management platform (TMS) 110, a 
signal transfer point (STP) 112, a manually-operated STP 
administrative interface 114, a network test message gen- 
erator/monitoring node 116 (e.g., the Tekelec MGTS™ 
testing and monitoring product), and a manually -operated 
administrative interface 118. It will be appreciated that STP 
112 and MGTS™116 are directly connected via dedicated 
signaling system 7 (SS7) signaling links 120. MGTS™116 
generates SS7 signaling message traffic and simulates SS7 
signaling nodes that could be connected to STP 112 in a real 
network. TMS 110 and the two administrative interface 
terminals 114 and 118 are connected to STP 112 and MGTS 
116 nodes via RS-232 links. More particularly, STP admin- 
istrative terminal 114 and MGTS™ administrative terminal 
118 are connected to the command line interfaces of STP 
112 and MGTS 116, respectively. TMS 110 is also connected 
to the command line interfaces of both STP 112 and MGTS 
116. 

[0008] As such, it will be appreciated that a complex test 
scenario could involve a large degree of operator interven- 
tion and subsequent manual configuration of the two devices 
under test. For instance, a particular test of STP 112 may 
require that MGTS™ 116 attempt to bring one of the plu- 
rality of SS7 communication links 120 into service. Execu- 
tion of this portion of the test can be initiated by TMS 110 
via one or more commands that are sent to the MGTS node 
via the TMS-to-MGTS™ command line interface connec- 
tion. A particular test scenario being executed by the TMS 
110 could require a dynamic or decision-tree-type response 
on the part of the TMS (i.e., selection of the next test is 
dependent on the result of the preceding test). For instance, 
TMS 110 may be required to perform a second test, 
TestCase_AlignedTrue if the particular SS7 signaling link in 
question was successfully aligned. If the signaling link in 
question was not successfully aligned, TMS 110 may be 
required to perform a third test, TestCase_AlignedFalse. As 
such, a human operator is required to observe the results of 
the first test via the MGTS™ administrative terminal 118, 
and provide feedback to TMS 110. Such manual feedback is 
human -resource-intensive, inefficient and prone to operator 
error. 
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[0009] With regard to overall test tool evolution, the 
incorporation of graphical user interface (GUI) based 
enhancements to testing systems have been instrumental in 
making the testing process more intuitive and user-friendly. 
However, such enhancements have not focused on the 
human operator-intensive elements of the testing process, 
such as reducing the need for device-specific commands and 
different terminals connected to each device being tested. 

[0010] Local area network (LAN) technology and com- 
munication architectures have also enhanced the testing of 
network products. GUIs are conveniendy located, either 
locally or remotely, to a particular data processing system 
that drives the actual testing of network products. Client/ 
server technology is utilized to provide remote access to test 
systems. A client application includes a GUI that allows a 
user to access a remote test system. A server application 
receives and processes requests from the client. The server 
also executes on a system in the network. The client/server 
framework allows a client to be located on any system in the 
network, even on the same system on which the server 
resides, 

[0011] With the many varieties of network products and 
methodologies for testing of network products currently 
available, test devices must be proficient in testing a large 
number of applications and executing the many varieties of 
test cases associated with those applications. Thus, a test 
device must be capable of learning or adapting to new test 
case formats and new test interfaces as required. 

[0012] A test case defines how a particular test will be 
performed. For example, a test case may simply be a batch 
file containing instructions for performing a test, or a test 
case may be a file of commands or directives administered 
through a GUI. Additionally, a test case may include a data 
structure stored in memory that is built by a data processing 
system upon receipt of instructions from a client or other 
application. Regardless of how a test case is implemented 
the, a test case embodies the action which will take place in 
order to perform a test. 

[0013] From an operational perspective, there exists a 
need for the capability for users in a network environment to 
share test cases and test case results for multiple devices 
under test in a test network environment. There also exists 
the need for a testing system that performs any type of test 
case without knowing the specific test case formats and 
methodologies of each and every test case available for 
execution on a test network. 

[0014] Additionally, there exists a need for the capability 
of designating the test functions to be tested as a sequence 
of interrelated, cross-communicated test functions. As dis- 
cussed briefly above with regard to the example shown in 
FIG. 1, values produced as output by a first DUT may be 
used as input by a second DUT. However, as indicated 
above, using the results from one test case as input to another 
test case requires manual intervention by an operator with 
detailed knowledge of device -specific test case commands. 
Accordingly, there exists a long-felt need for methods and 
systems for testing communications network components 
that reduce the need for operator intervention and knowl- 
edge of device-specific test commands. 

DISCLOSURE OF THE INVENTION 
[0015] Hie present invention includes methods and sys- 
tems for creating and executing sequences of interrelated 



test cases and providing a generalized test environment that 
allows complete automation of test cases. The test cases are 
independent of the number or types of devices under test. As 
such, a test operator responsible for writing a test script need 
not know the device-specific commands because test envi- 
ronment device packages map the device-specific com- 
mands to a common script language. Therefore, the operator 
need only be familiar with a common script language rather 
than device-specific test commands for multiple devices 
under test. Furthermore, the present invention provides a 
common interface that allows interaction between multiple 
devices under test, thereby achieving the desired system 
performance described above. 

[0016] According to one aspect, the present invention 
includes a communications network testing system that 
facilitates autonomous or attendant-free interaction between 
the administrative interfaces of multiple network devices 
under test. The present invention includes an abstract com- 
mand language (ACL) interface that enables cross-commu- 
nication between dissimilar devices (i.e., devices that 
employ different administrative interface protocols). As 
such, the results of a first test on a first device under test may 
be used to automatically trigger the reconfiguration of a 
second DUT and subsequent execution of a second test. 
Furthermore, the second test may utilize information 
returned from the first test on the first DUT. Again, it is 
particularly significant that the first DUT triggers a recon- 
figuration of the second DUT, even though the first DUT and 
second DUT may employ different administrative interface 
protocols. Such functionality allows complex test sequences 
that involve interaction among multiple DUTs to be easily 
automated and remotely executed without requiring a sig- 
nificant degree of human test operator intervention. The 
abstract command language interface has a layered archi- 
tecture that includes a tool command language (TCL) 
object-based interface layer. This TCL object-based layer 
utilizes the abstract command language to facilitate com- 
munication with various independent testing and switching 
equipment via access to their respective administrative com- 
mand line interfaces. A unique, device-specific communica- 
tion interface package is included for each type of device 
under test. 

[0017] According to another aspect, the present invention 
includes a fully integrated test case and test plan editor, 
where test plans and their associated test cases are main- 
tained in a version-controlled environment. 

[0018] According to yet another aspect, the present inven- 
tion includes an arbitration engine that dynamically allocates 
resources when users attempt to simultaneously schedule 
test cases that require the same resources. 

[0019] The version-controlled environment may be 
extended to include operating software associated with each 
DUT. As such, a test plan may specify a particular version 
of a test case, which may in turn specify a particular program 
or software version that is to be installed on a given DUT for 
the duration of the test. Corresponding test results may also 
be stored and maintained in the version-controlled environ- 
ment. 

[0020] The present invention further includes a graphical 
user interface that may be accessed and operated via a wide 
area network or local area network connection. This GUI 
simultaneously displays test plan, test case, and correspond- 
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ing test results. The GUI is further adapted to provide access 
to the test plan/test case editor, as described above. 

[0021] Accordingly, it is an object of the present invention 
to provide a test system that facilitates real-time, inter-DUT 
communication during execution of a test without requiring 
human test operator intervention. 

[0022] It is another object of the present invention to 
provide communication interface packages for mapping 
abstract command language commands to device -specific 
commands so that a test case designer is not required to have 
specific knowledge of device -specific commands. 

[0023] Some of the objects of the invention having been 
stated hereinabove, other objects will become evident as the 
description proceeds, when taken in connection with the 
accompanying drawings as best described hereinbelow. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0024] FIG. 1 is a schematic diagram illustrating a con- 
ventional test system architecture. 

[0025] FIG. 2 is a schematic diagram illustrating a test 
system according to an embodiment of the present inven- 
tion. 

[0026] FIG. 3 is a schematic diagram illustrating a test 
environment that utilizes a test system according to an 
embodiment of the present invention. 

[0027] FIG. 4 is a block diagram illustrating a test man- 
agement system client and version control system according 
to an embodiment of the present invention. 

[0028] FIG. 5 is a block diagram illustrating exemplary 
contents of a communication interface package according to 
an embodiment of the present invention. 

[0029] FIG. 6 is a block diagram illustrating an execution 
manager according to an embodiment of the present inven- 
tion. 

[0030] FIG. 7 is a flow chart illustrating exemplary func- 
tional processes or steps associated with test system opera- 
tion according to an embodiment of the present invention. 

[0031] FIG. 8 is a block diagram illustrating exemplary 
test management system input and output messages accord- 
ing to an embodiment of the present invention. 

DETAILED DESCRIPTION OF THE 
INVENTION 

[0032] FIG. 2 is a simplified schematic diagram of a test 
system according to an embodiment of the present inven- 
tion. In FIG. 2, test system 200 includes a test tools server 
210, a browser 212, a test management system client 214, 
and a test controller 215, all of which are connected via 
network 216. Server 210, browser 212, client 214 and 
controller 215 may each execute on general purpose com- 
puting platforms. Network 216 may be a private wide or 
local area network (i.e., corporate intranet) or a public WAN, 
such as the Internet. 

[0033] Test tools server 210 maintains a library of device- 
specific communication interface packages, where each 
device-specific communication interface package contains 
information for mapping executable instructions between a 
common or abstract command language and the command 



line interface language of a specific device. Test tools server 
210 may also maintain one or more generic communication 
interface packages that provide certain basic, core commu- 
nication interface functionality that is device independent. 
Exemplary communication interface packages will be dis- 
cussed in detail below. 

[0034] In one embodiment, test tools server 210 maintains 
the communication interface package library in a version- 
controlled environment, such as that provided by the 
CLEARCASE™ available from Rational Software Corpo- 
ration or similar version control system. As such, multiple 
versions of a single device-specific communication interface 
may be maintained in a secure and version-controlled man- 
ner. 

[0035] In addition to maintaining a communication inter- 
face package library, test tools server 210 may also maintain 
a library of test plans and the associated individual test cases 
used in formulating test plans. In one embodiment, such test 
cases may include data files containing tool control language 
and abstract command language instructions. In any event, 
such test plan and test case information is maintained in a 
version-controlled environment, in much the same manner 
as the communication interface package library described 
above. Following the execution of a test plan or individual 
test case, test tools server 210 maintains the associated test 
results in a test results library that preferably also resides in 
a version -controlled environment. As such, all aspects of a 
particular test scenario (e.g., communication interface pack- 
ages, test plans, test cases, and test results) are maintained in 
a version-controlled environment, which significantly 
enhances the quality control and auditing process associated 
with communication component and system testing. 

[0036] Browser 212 allows a user to simultaneously view 
test plans, test cases and corresponding test results associ- 
ated with various systems of devices or individual devices 
under test. As described above, such information is main- 
tained in a version-controlled environment on test tools 
server 210 and is accessible via communications network 
216. Browser 212 may execute on a disk operating system 
(DOS), WINDOWS®, or Unix-based computing platform 
Examples of browsers suitable for use with embodiments of 
the present invention include Netscape NAVIGATOR® or 
Microsoft INTERNET EXPLORER®. Aplurality of brows- 
ers can be simultaneously connected to test tools server 210 
via communications network 216 for simultaneous viewing 
of test cases and test case results. In the case where com- 
munications network 216 corresponds to a public WAN, 
such as the Internet, browsers 212 may execute on comput- 
ers geographically located at great distances from the actual 
test laboratory where the physical test equipment is located. 
A test system of the present invention facilitates the testing 
of a communications system that includes multiple DUTs 
where the DUTs are not physically co-located. 

[0037] Test management system client 214 communicates 
with test tools server 210 via network 216 and allows a user 
to perform a number of activities associated with test system 
operation including: scheduling test plan execution, con- 
structing and editing of test plans, and constructing and 
editing test cases. Because the test plan, test case, and 
communication interface package information is maintained 
in a version-controlled environment on test tools server 210, 
multiple test management system clients or client sessions 
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may be simultaneously active and ia communication with 
test tools server 210 without the risk of operator confusion 
due to version problems. 

[0038] FIG. 3 illustrates a test tools server 210, a test 
management system client 214, and a test controller 215 
connected to multiple devices under test. In one embodi- 
ment, such devices under test may include, but are not 
limited to: a service control point (SCP) 218, a home 
location register (HLR) 219, a visitor location register 
(VLR) 220, a short message service (SMSC) 221, a domain 
name system (DNS) server 222, a data packet router 224, a 
signaling gateway (SG) 226, a network simulator and moni- 
toring device 228, a signal transfer point 229, and a soft- 
switch (SS) node 230. The communication nodes listed 
above are well known to those skilled in the art of telecom- 
munication and data networks, and as such will not be 
described in detail herein. A detailed description of the 
illustrated signaling system 7 (SS7) and IP communications 
devices (e.g. SCP 218, SG 226, STP 229, and Softswitch 
230) can be found in Russel, Travis, Signaling System #7, 
McGraw-Hill, Third Edition, 2000. A detailed description of 
the illustrated wireless communication network elements 
(e.g. HLR 219, VLR 220, and SMSC 221) can be found in 
Mouly, M. and Pautet, M. B,, The GSM System for Mobile 
Communications, Cell & Sys, First Edition, 1992. Data 
communications network elements 222 and 224 are 
described in Stevens, Richard, TCP/IP Illustrated, Volume I: 
The Protocols , Addison-Wesley, 1994. 

[0039] In order to test one or more devices, test manage- 
ment system client 214 accesses a test plan and associated 
test case files from test tools server 210 and requests that the 
test cases be executed by test controller 215. Test controller 
215 establishes connections with the devices under test and 
executes the test cases received from test tools server 210. 
Results associated with a particular test plan or test case are 
then provided to test tools server 210 for storage in the 
version-controlled environment. According to an important 
aspect of the invention, test tools server 210 includes an 
arbitration engine 231 that dynamically allocates resources 
when users attempt to simultaneously attempt to use the 
same resources. The automated test system architecture 
according to the present invention allows multiple users in 
multiple locations to utilize the same test system resources. 
For example, a test system user in Richardson, Tex. and a 
test system user in Morrisville, N.C. may each desire to test 
software on STP 229, which is located in a lab in Morris- 
ville, N.C. Both users may access test toots server 210 using 
TMS clients 214 and schedule tests to be executed. Test tools 
server 210 may include a static scheduler that allows the 
users to select time slots for test execution in an attempt to 
avoid scheduling conflicts. However, because test cases can 
run longer than expected, one user's test may conflict with 
the time slot scheduled for another user's test. Arbitration 
engine 231 dynamically allocates resources, in this case, 
STP 229, so that the test cases do not conflict with each 
other. In this example, if user A's test case is nearly 
complete, arbitration engine 231 may allow the test to 
complete and dynamically re-schedule the test requested by 
user B. Exemplary dynamic resource allocation algorithms 
that may be used by arbitration engine 231 include random 
allocation, first in, first out, priority scheduling, etc. Using 
one or more of these resource allocation algorithms, arbi- 
tration engine 231 dynamically resolves resource conflicts 
before they occur. 



[0040] As illustrated in FIG. 3, test controller 215 is 
capable of simultaneously communicating with multiple 
DUTs. Test controller2l5 also facilitates autonomous or 
attendant-free interaction between the administrative inter- 
faces of multiple network devices under test. As such, the 
results of a first test on a first device under test may be used 
to automatically trigger the reconfiguration of a second 
device under test and the subsequent execution of a second 
test, where second test may utilize information returned 
from the first test on the first device under test. 

[0041] It is an important aspect of the invention that test 
controller 215 allows a first DUT to trigger a reconfiguration 
of a second DUT, even though the first DUT and the second 
DUT may be completely different types of communication 
devices with different administrative or command line inter- 
faces. Such functionality allows complex test sequences that 
involve interaction among multiple DUTs to be easily auto- 
mated and remotely executed without requiring a significant 
degree of human test operator intervention. For instance, in 
the case of SS7 telecommunications related devices that 
utilize the message transfer part (MTP) protocol, complete 
MTP level two or level three conformance testing can be 
accomplished in a matter of hours with little or no human- 
operator intervention. Previously, such comprehensive test- 
ing was human-operator-intensive and could take one or 
more weeks to complete. 

[0042] FIG. 4 is block diagram of one embodiment of a 
test system of the present invention including a TMS client 
214, a test controller 215, a system of devices under test 302, 
and a test tools server 210. In this embodiment, TMS client 
214 includes a graphical user interface 310, a test plan/test 
case editor 314„ and a reporting manager 318. Test control- 
ler 215 includes a test plan and test case execution manager 
for executing test cases received from test tools server 210. 
GUI 310 provides a human operator with an easy-to-use, 
icon-driven interface to the overall test system of the present 
invention. GUI 310 enables an operator with minimal 
knowledge of the devices under test and their respective 
administrative interface protocols to perform a complex 
battery of integration and system performance tests. 

[0043] Test tools server 210 illustrated in FIG. 4 includes 
a version control environment 350, such as CLEAR- 
CASE™. A number of database elements, including a test 
plan/test case library 352, a test results library 354, a 
communication interface package library 356, and a device 
under test operating software library 358, reside within 
version-controlled environment 350. Individual software 
files (e.g., source code, compiled code, scripts, data, etc.) are 
stored in such an environment so that user access to and 
manipulation of the files may be controlled and monitored. 

[0044] According to an important aspect, the present 
invention includes a plurality of communication interface 
packages. A communication interface package is a software 
entity that defines the mapping between device-specific 
command line interface commands and common or device- 
independent abstract command language commands. In a 
preferred embodiment, communication interface packages 
include classes having functions and data structures used to 
test specific devices. Examples of mapping functions that 
may be included in a communication interface package are 
shown in FIG. 5. In FIG. 5, communication interface 
package 370 includes mapping information associated with 
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a signaling gateway (SG). The term "signaling gateway," as 
used herein, refers to a network element capable of routing 
both SS7 and IP messages. In this particular example, an 
abstract command language command "IsLinkAligned(L,l, 
1)" that is related to the alignment status of a communication 
link is mapped to the corresponding signaling-gateway- 
specific command line interface command, "REPT-STAT- 
SLK:LOC=1101:PORT«A." Functionally similar abstract 
command language commands associated with different 
devices may utilize the same root command with different 
arguments. For instance, a communication interface package 
associated with a service control point may include the 
following link alignment status abstract command language 
command, "IsLinkAligned(L5)," where L5 is a signaling 
link connected to the service control point. As such, different 
devices can share a command structure that is common to a 
large degree, and highly intuitive from an operator's per- 
spective. 

[0045] Referring back to FIG. 4, communication interface 
packages may be stored in communication interface package 
library 356 that is maintained in version-controlled environ- 
ment 350. As discussed above, communication interface 
packages may include software classes created using an 
object-oriented programming environment, such as C++ or 



J++. Consequently, a new communication interface package 
object can be created which inherits some or all of the 
properties associated with a parent communication interface 
package object. This inheritance capability facilitates the 
rapid creation of communication interface packages as new 
devices and new versions of existing devices are added to a 
test system. Communication interface packages may be 
included and compiled in text cases using INCLUDE state- 
ments or other statements that allow files to be incorporated 
into the compiled version of a test case. 

[0046] Test plan and test case editor 314 allows a user to 
create and edit individual test cases as well as high-level test 
plans. In the most simplistic embodiment, a test case is a text 
file that contains a sequence of test instructions for testing 
one or more devices under test. A test plan is a sequence of 
test cases. Such test instructions may include abstract com- 
mand language and tool command language commands that 
perform specific operations on a DUT. These instructions 
may also facilitate conditional processing based on the 
results returned from one or more devices under test. In 
addition, multiple test cases can be combined into a single 
test case data file and thereby provide test-plan-like func- 
tionality. A sample test case is shown below: 



package require Mgts 
package require Eagle 
package require titen 

# This is the startup code that wold be run before any test 

# purposes are started 
proc startup {}{ 

Mgts mgtsl $::env(MGTS_SHELF) $::env(MGTS„BUILD) 
Eagle eaglel $ ::env(EAGLE_tP) $:: env{EAGLE_PORT) 
eagle login $::env(EAGLE_LOGIN) 
$::env(EAGLE_PASSWORD) result 

} 
# 

# This is the cleanup code that is run after all the test purposes 

# have completed execution, 
proc cleanup {}{ 

Mgts::deleteAll 
Eagle: :deletcAll 

} 

# Test Purpose #1 
step tp { 

Purpose - ANSI Level 3 AQTcst7.2.4 

>{ 

set nodename $:xnv(L3_NODE„NAME) 
set Ll_l_card $::env(Ll_l_CARD) 
set Ll_l_port $::env^l_l_PORT) 
infoline "Start the State Machine. L3 Node Name is 
Snodename" 

mgtsl ranSM Snodename AQTest7-2-4 
Mgts::expectEvent { 

MGTSState Transition^, V,UserAction){ 

# Got User Action 

infoline "Inhibiting link U__l" 
break 

} 

MGTStatc Transilion( V,*,NotAligncd){ 

# The State Machine did not align. *DO NOT* continue 
infoline "Transitioned through NotAligned state!" 
result ABORTED 

return 

} 

} 

# Now, inhibit link L1_J 
eaglel Iiihibit_Link(LU) 

if {![Eagle::isOK result]} { 
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-continued 



infoline "Eagle rejected command, check LOC and PORT 
variables $result(command)'* 

result ABORTED 
return 

} 

Mgts::expectEvent{ 

MGTSState Transition^, • ,FAILED){ 

# One of the State Machines failed. *DO NOT* continue 
info line "Transitioned through FAIL state!" 

result FAIL 
return 

} 

MGTState Transiiion(*, V>PASSED){ 

# State Machines Passed 

infoline "Transitioned through PASS state" 

result PASS 

return 

} 

MGTState Transition^ /, User Action,S top) { 
infoline "Transitioned through Timeout state! Failing 

tes tease." 

result FAIL 
return 

} 

} 

} 

# Start execution 
run_testcase 



[0047] In the code example, the "package require 5 ' state- 
ments include device packages required for the script. In this 
example, the device specific packages that are included are 
mgts, Eagle, and Titen. According to an important aspect, 
the Titen package includes procedures that allow the user to 
create test cases using generic commands. The procedures 
access the device specific packages for multiple devices 
being tested and perform the functions for each specific 
device. For example, the "proc startup {}" procedure 
includes code for starting up and logging into each device. 
The procedure also creates an instance of an object for each 
device. For example, in the statement "MGTS mgtsl, 
""MGTS" creates an instance of an MGTS object having an 
identifier of "mgtsl." The remaining statements in the "proc 
startupj}" procedure control the login and password for the 
Eagle device. Thus, all the user is required to do in writing 
a test script is specify the command "proc startup{ }" and the 
devices that the user desires to start. The code in the 
"proc_startup{} procedure accesses the device packages for 
the individual devices and performs the functions, such as 
initializing memory, bringing links into service, aligning 
links, etc., for each specific device. 

[0048] The "proc cleanup {}" procedure includes cleanup 
routines for each device that are automatically called at the 
end of the script. Like the "startup{}" procedure, the cleanup 
procedure is a procedure in the titen package that accesses 
the individual device packages. The "cleanup{}" procedure 
performs device-specific cleanup functions, such as freeing 
memory, disabling links, etc. Again, all the test case designer 
is required to remember is the "cleanup {}" command and 
the names of the devices that the designer desires to shut 
down. Thus, the titen package reduces the time and effort 
required for test case design. 



[0049] The "step tp" statement defines a test in a test case. 
The steps are counted automatically by the Titen package 
and a result for each step will be generated if using TET/ 
API-compliant test cases. 

[0050] The "purpose" statement is a free form text field 
describing a test case. This statement can be a block of text 
that describes the execution of a test. 

[0051] The "set" statements set environment variables 
read into the test. 

[0052] The "mgtsl runSM" statement runs the name or 
number of the desired state machine on the specified node. 
In this example, the state machine is "AQTest7-2-4." 

[0053] The "Mgts::expectEvent" procedure looks for the 
state machine to advance to a particular state in the PASM 
map. The UserAction state looks for this state in PASM and 
indicates that a used action on the device under test is 
needed. 

[0054] The "Irihibit_Link" command is an abstract com- 
mand language command which may be converted into a 
device specific command, such as eagle cli "INH-SLK- 
:LOC=$Ll_l_card:PORT=$Ll_l_port". In this example, 
the abstract command language command is converted into 
an Eagle-specific command including specific variables to 
inhibit a signaling link of an Eagle STP. 

[0055] The "Mgts: :expectEvent" procedure looks for the 
PASM state transition after the Eagle command was sent. In 
particular, the procedure looks for FAILED, PASSED, or a 
transition from a userAction state to Stop for a timeout or 
abort. 

[0056] The "run_testcase" statement is a wrapper that calls 
execution of the test case. 

[0057] Test plan and test case editor 314 communicates 
and manages interaction with the version-controlled test 
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plan and test case library 352. As such, individual test cases 
and test plans are checked in and out of version control 
environment 350 via editor 314. Editor 314 is also adapted 
to interface with the graphical user interface 310, thereby 
providing users with an easy to use and intuitive file cre- 
ation/modification environment. 

[0058] As stated above, test tools server 210 includes a 
scheduler that allows the user(s) to schedule immediate or 
delayed execution of individual test cases or test plans. Such 
test case and test plan scheduling capability enables test 
operators to efficiently allocate high-demand resources in a 
test lab environment by scheduling the execution of a test 
plan or test case during off-hours or periods of low resource 
usage. In one embodiment, the scheduler allocates resources 
by examining test environment requirements. In the ideal 
situation, the test environment only specifies the type of 
resource the test requires, and it is the responsibility of the 
scheduler to allocate the necessary resource(s) prior to test 
case execution. This approach works best for environments 
with a large number of uniform resources. Such a typical 
case would be an E-commerce server where the resource is 
the login ID of a particular customer or large-scale telecom- 
munications equipment where the test is restricted to a 
particular shelf or even a card within a shelf and no special 
wiring is required (i.e., in an environment where the ratio of 
resources required by a test case to total number of generic 
resources is very low). On the other hand, if the number of 
resources used by a test case is close to the total amount of 
available resources, it may be more efficient to schedule on 
per-resource basis. Such is often the case in a typical 
telecommunications lab where equipment cannot be used to 
run more than one test case at any time, and there are only 
a few units available. In such a case, the resource schedule 
is defined as a relation between instances of resources, tester, 
test suite and time interval. Resource schedules are stored in 
a resource schedule repository accessible by the scheduler. 

[0059] In any event, manager 316 receives test plan and/or 
test case execution instructions from a test operator via GUI 
310 and subsequently extracts or reserves the required test 
plan or test case files from test plan or test case library 352 
and communication interface package files from communi- 
cation interface package library 356, both of which are 
maintained in version control environment 350. Once all 
appropriate test plan, test case and communication interface 
package files have been obtained from version control 
environment 350, the indicated test or tests are executed. 
The results of a particular test plan or test case execution are 
subsequently stored in a test results library 354, which is 
also maintained in version-controlled environment 350. 

[0060] In one embodiment, test plan and test case manager 
316 examines information contained in a test case or test 
plan related to the operating software version required on a 
particular test device. Once such operating software version 
information is determined, manager 316 communicates with 
a device under test operating software library 358 contained 
in the version control environment 350. DUT operating 
software library 358 maintains software associated with 
particular test devices (e.g., generic program loads, operat- 
ing systems, feature enhancements, etc.), which enables a 
test system of the present invention to automatically and 
dynamically configure a DUT using any number of pre- 
defined software loads. As such, a particular version or 
revision of operating software associated with a DUT can be 



automatically downloaded and initialized on a specific DUT 
that is involved with the test plan/test case being executed. 

[0061] Reporting manager 318 is extracts some or all of 
the files associated with a test plan or test case from version 
control environment 350. These files may include test plan 
or test case files, associated CIP files, test result files, and 
DUT operating software files. As such, reporting manager 
process 318 facilitates the presentation of a comprehensive, 
integrated view of information related to a particular test 
plan or test case including: test plan or test case instructions, 
relevant DUT specific command packages, results associ- 
ated with the executed test plan/test case instructions, and 
operating software source code associated with relevant 
DUT(s). This information is accessible and displayed to a 
test operator via GUI interface 310. 

[0062] Test Plan/Case Execution Engine 

[0063] FIG. 6 illustrates one embodiment of test plan and 
test case execution manager 316, including information flow 
pathways associated with a sample system under test. The 
functional blocks illustrated in FIG. 6 are intended to 
illustrate services provided by execution manager 316. Prac- 
tical implementation of this functionality could be accom- 
plished using a variety of software constructs and architec- 
tures, and as such, the block diagram shown in FIG. 6 
should be considered as only one of many possible imple- 
mentation schemes. 

[0064] In addition to execution manager 316, FIG. 6 
includes a system under test that includes a signaling gate- 
way node 226, a message generator and traffic simulation 
device 228, and a Softswitch 230. The functions provided by 
each of these devices is not particularly relevant to the 
operation of the present invention, and as such detailed 
descriptions of these devices are not provided herein. For the 
purposes of illustration, each of the devices under test are 
assumed to include a command fine interface with which the 
execution manager 316 communicates. However, it will be 
appreciated that in an alternate embodiment of the present 
invention, execution manager 316 may communicate with a 
device under test that utilizes a GUI-type interface as 
opposed to a command line interface. In such cases, a 
GUI-interfaced device under test may be configured to serve 
as a remote host and accept command inputs from the 
execution manager 316 via a connection that is similar in 
function to that provided by commercially available remote 
access products such as PCANYWHERE™ available from 
Symantek Corporation or CARBONCOPY™ available 
from Compaq Corporation. In an alternate embodiment of 
the invention, remote access can be achieved using any 
TET-enabled remote client, such as TestExpert, available 
from Lumenar, Inc., that supports client/server on the execu- 
tion host or on a thin client web browser The commands and 
command structures necessary to communicate with such 
GUI driven devices would be accommodated within the 
basic architecture of the present invention via GUI -cap able 
communication interface packages. 

[0065] Once again, for the purposes of illustration, the 
detailed discussions presented herein are limited to commu- 
nication with those devices that employ command line 
interfaces, since this type of interface is currently the most 
widely deployed interface type. Communication with GUI- 
based devices can occur via a graphical user interface if a 
suitable GUI tester is added via a new package. Returning to 
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FIG. 6, execution manager 316 includes an execution 
engine 400, a titen package 402, a plurality of device- 
specific test case packages 404, and a communication inter- 
face layer 420. Execution engine 400 reads, interprets, and 
executes commands in a test plan or test case file. Titen 
package 402 includes generic classes for executing test 
cases, recording results, and performing common functions 
by accessing individual device packages, as discussed 
above. Packages 406, 408, 410, and 412 include device- 
specific classes. According to an important aspect of the 
invention, these device-specific classes include functions 
that translate generic commands to device-specific com- 
mands. In one embodiment, the device -specific commands 
may be tool command language commands. In an alternate 
embodiment, the device-specific commands may be imple- 
mented using an alternate language, such as PERL, JAVA, or 
C++. The abstract-command-language-to-tool-command- 
language command mapping information is obtained for any 
given DUT from a DUT specific communication interface 
package. As described above, FIG. 5 illustrates exemplary 
tool command language to DUT-specific commands, 

[0066] In the embodiment shown in FIG. 6, a DUT 
package is included for each DUT that is involved in a test. 
Each device-specific package includes device-specific com- 
mands and mappings from generic to device-specific com- 
mands. In the example shown in FIG. 6, the device-specific 
packages include a signaling gateway package 406, a soft- 
switch package 408, and a message generator/traflic simu- 
lator (MGTS™) package 410. 

[0067] Titen package 402 is the "glue" software compo- 
nent that pulls all of the packages together and provides 
basic common environment functionality. It is the part of the 
present invention that interfaces to the high-level TMS to the 
scripting languages. For example, titen package 402 handles 
automatically counting the steps of a test case and assem- 
bling them into the invocable components required by the 
test environment toolkit (TET). The TET is a universal 
management and reporting framework for conformance 
tests. Further information regarding TET can be found at 
www.tetworks.opengroup.org. Uten package 402 also 
includes start-up and clean-up routines required by the 
device packages. This allows concise, clean, and easy means 
to handle all device package overhead. Thus, all of the setup 
and clean up on the main script can be done with a couple 
of lines of code. 

[0068] In one embodiment, titen package 402 provides the 
functions step, invocableComponent, infoline, result, run_t- 
estcase, try, startup, and cleanup. These actions performed 
by these functions are as follows: 

[0069] step: 

[0070] a command used to define a step in a testcase, 
which can then be automatically counted. 

[0071] invocable Component, subTest: 

[0072] a command used to define a new TET invo- 
cable component. Each invocable component con- 
tains a set of test case steps. 

[0073] infoline: 

[0074] a command used to print output into a journal 
file. 



[0075] result: 

[0076] a command used to report the result of a step. 
If no result is provided, it defaults to NoResult. 

[0077] run_testcase : 

[0078] a command that is a wrapper to call for 
execution of the testcase 

[0079] try: 

[0080] a command similar to JAVA'S "try" command 
that evaluates the expressions given 

[0081] startup, cleanup: 

[0082] initialization and cleanup for execution of 
packages 

[0083] The command set listed above are procedures 
defined in the titen package. Some of the procedures, such 
as startup and cleanup, greatly reduce test case complexity 
because they include code that accesses the individual 
device packages for performing startup and cleanup func- 
tions. For example, if a user specifies "startup eaglel, 
routerl," the startup procedure in the titen package auto- 
matically accesses the STP and router packages, and deter- 
mines how to start each device. The test case designer need 
not be familiar with individual device commands or startup 
procedures. Allowing actions to be performed on multiple 
devices using a single generic command is an important 
feature of the invention. 

[0084] In the embodiment shown in FIG. 6, communica- 
tion interface 420 automatically establishes a telnet session 
with the command line interface of each device under test. 
The present invention is not limited to using telnet to 
connect to the command line interface of the device under 
test. Any protocol that allows remote terminal access to a 
machine is intended to be within the scope of the invention. 
In any event, communication interface 420 is further adapted 
to send and receive DUT-specific command line interface 
commands and receive DUT-specific results. As indicated in 
FIG. 6, such commands are generated by execution engine 
400, while DUT test results are generated by the various 
devices that comprise the system under test (i.e., SG 226, SS 
230, and MGTS™228). 

[0085] Test Plan and Test Case Execution 

[0086] Continuing with the example test scenario pre- 
sented in FIG. 6, SG 226 and MGTS™228 communicate via 
signaling system 7 (SS7) signaling links, while SG 226 and 
SS node 230 communicate via stream control transmission 
protocol/Internet protocol (SCTP/IP). It should be noted that 
the test system of the present invention is not concerned with 
nor adapted to communicate with test devices via such 
inter-nodal communication interfaces. Instead, as discussed 
above, the present invention communicates with and con- 
trols devices under test via their administrative interfaces 
(e.g., command line, GUI). 

[0087] FIG. 7 is a process flow diagram that describes 
exemplary steps associated with execution of a test case 
according to an embodiment of the invention. Correspond- 
ing information flow pathways are indicated in FIG. 6. In 
the description below, it is assumed that a particular test case 
has been selected for execution by a test operator via a TMS 
client 214. Test tools server schedules the test for execution 
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and when the time for execution arrives, extracts the test 
from library 352 and downloads the test to execution engine 
400(ST1). Execution engine 400 opens the file and begins 
reading the file. Information contained within the test case 
file identifies one or more of the devices under test. Execu- 
tion engine 400 accesses communication interface package 
library 356 and extracts the appropriate communication 
interface packages associated with each DUT t as indicated in 
step ST2. Information may also be included in the test case 
file that identifies particular DUT operating software 
requirements associated with the test, in which case execu- 
tion engine 400 accesses DUT software library 358 and 
directs the appropriate software loads to each DUT (ST3). 

[0088] Once initialization of all DUTs is complete and all 
appropriate DUT objects and instances have been created, 
test command instruction processing begins. The execution 
engine 400 reads an abstract command language command 
(ST4) and, based on the mapping provided by the appropri- 
ate communication interface package, interprets the com- 
mand within the context of the specific DUT to which the 
command refers (ST5). For instance, the abstract command 
language command "IsLinkAligned(Ll,l)," when directed 
to SG 226, would require input from SG package 406. The 
SG-specific communication interface package that was 
obtained from CIP library 356 contains the abstract-com- 
mand-language-to-tool-command-language mapping infor- 
mation necessary for execution engine 400 to successfully 
interpret the command and produce an equivalent tool 
command language command (ST6). It should be noted that 
the mapping of abstract command language commands to 
tool command language commands does not necessarily 
yield a one-to-one correspondence. That is, a single abstract 
command language command associated with an SG node 
may map or translate to a multi-command tool command 
language sequence. The same property holds true for tool 
command language to abstract command language com- 
mand mapping. That is, multiple abstract command lan- 
guage commands may correspond to a single tool command 
language command. Thus, the present invention is not 
limited to a one-to-one between abstract command language 
commands and tool command language commands. 

[0089] In any event, the resulting tool command language 
command is subsequently passed to the communication 
interface 420 (ST7), as indicated in FIG. 6. In one embodi- 
ment, communication interface 420 contains components 
that include commonly available software tools, such as the 
Expect tool, available from http://expect.nist.gov. Software 
tools, such as Expect, provide the communication interface 
420 with the ability to establish telnet sessions with various 
devices under test, and also provide a means for capturing 
streams of response data (e.g., ASCII character data) from a 
DUT. In the case of response data, once captured this data 
is provided to execution engine 400 for subsequent process- 
ing. 

[0090] As such, the outbound command is transmitted via 
a telnet session to the appropriate DUT (ST8). An example 
of one such outbound signaling gateway command 372 is 
shown in FIG. 8. Command 372 illustrated in FIG. 8 
requests the status of a signaling link connected to signaling 
gateway 226. Communication interface 420 may subse- 
quently enter a "wait" mode, during which interface 420 
monitors the telnet session and waits for a response from the 
DUT (ST9). Once again, a typical DUT response is com- 



prised of a stream of ASCII characters that form a text 
message which was originally intended to be read by a 
human operator via the DUTs administrative interface. 
Such DUT response information might include test result 
information, device configuration information, device status 
information, an acknowledgment message associated with a 
received command, etc. An example of one such response 
message 374 is shown in FIG. 8. In FIG, 8, response 
message 374 indicates a state transition for a signaling link 
from active to aligned. 

[0091] In one embodiment, communication interface 420 
intercepts the response data stream and provides execution 
engine 400 with the DUT response message in ASCII text 
format. Execution engine 400 receives the DUT response 
message, parses, and examines the text message (SIU). The 
ability to receive and parse DUT response messages without 
human operator intervention is one of the key aspects of the 
present invention. Such functionality enables test case 
scripts to be created that are capable of employing branching 
or conditional code structures that effectively enable 
dynamic test sequences to be executed which vary depend- 
ing on the response of a DUT to a previous command (ST12 
and ST13). For example, a test case can be created which 
directs MGTS™ device 228 to attempt to establish or case 
code could be structured such that if a response message is 
received from MGTS™228 indicating that the link could not 
be successfully aligned then a second link alignment should 
be attempted. However, in the event that a response message 
is received from MGTS™228 indicating that the link was 
successfully aligned then the MGTS™228 is directed to 
begin generating and transmitting a sequence of test SS7 
message signaling units (MSUs) to SG 226. 

[0092] An even more powerful extension of this particular 
test system functionality involves the ability of a first DUT 
to control a second DUT, even though the first DUT has no 
knowledge of the second DUT's control or administrative 
interface protocol. Such functionality is accomplished in 
much the same manner as is described in the previous 
example, only in this case the branching or conditional 
construct in the test case code involves multiple DUTs, 

[0093] For example, a test case can be created which 
directs MGTS™ device 228 to attempt to establish or align 
a first SS7 signaling link that would be connected to SG 
node 226. The test case code could be structured such that 
if a response message is received from MGTS™228 indi- 
cating that the first link could not be successfully aligned 
then another alignment attempt on the first link should be 
initiated. However, in the event that a response message is 
received from MGTS™228 indicating that the first link was 
successfully aligned, then SG 226 is subsequently directed 
to drop or terminate a second signaling link. As such, 
MGTS™228 effectively directs SG 228 to initiate a link 
termination action. 

[0094] The following code example illustrates a portion of 
a test case in which execution engine 400 simultaneously 
tests multiple devices. In this example, the first device under 
test is an STP, referred to as "eagle." The second device 
under test is an IP router, referred to in the code as "routerl." 
Finally, a message generator/traffic simulator device 
"mgtsl" is used to perform various tests on the devices 
under test. The line numbers appearing on the right hand side 
of each line of code are included for illustrative purposes and 
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will be used to explain the functions of various lines of code. 
In the lines of code, the statements that begin with # signs 
are comments. 



1 # deactivate SS7 link and verify alarm 

2 # else deactivate IP router port and verify alarm in 3 sec 
3 

4 mgtsl cli Deactivate link: Port- $SSPl_port 
5 
6 
7 
8 
9 

10 
11 
12 
13 
14 

15 } else { 

16 # deactivate an IP Router port" 

17 router 1 cli Deactivate port: PorU= $SSP2__port 

18 infcline "Router IP port disabled" 

19 } 

20 sleep 1 

21 set time [expr Stime +1] 

22 } 



[0095] In line 4 of the code, execution engine 400 deac- 
tivates SSP port 1 on the MGTS. In a real network, deac- 
tivation of a port from an SSP should trigger an alarm at the 
corresponding STP. Accordingly, in line 8 of the code, the 
test case includes a while loop that checks for an alarm at the 
eagle STP. In line 9 of the code, the test case checks for the 
alarm status of the eagle. In line 10, the code checks for 
whether there is a major alarm on the eagle. In line 11, the 
code sets the eagle alarm flag if a major alarm occurs. Line 
12 of the code logs the alarm in a log file. I 

[0096] In line 17, the code deactivates SSP port 2 on an IP 
router. In line 18, the code logs the fact that the IP router port 
was disabled. The code then returns to the beginning of the 
while loop and checks whether the Eagle detects the fact that 
the IP router port to the SSP was disabled. Thus, in the code 
example, the execution manager performs a conditional 
action based on data received from a device under test. This 
is a feature of the invention that has not previously been 
possible in conventional test systems. In conventional test 
systems, a human operator would be required to enter the 
command for deactivating a link via the command line 
interface of one device, monitor the results on a separate 
terminal, decide what action to take based on the results, and 
manually reconfigure the second device. Accordingly, it can 
be seen that the present invention can save time and effort 
when testing multiple different devices. 

[0097] It will be appreciated that significantly more com- 
plex multi-DUT interactions can be easily accommodated by 
a test system of the present invention. For instance, SS7 
level 2 and level 3 conformance testing is currently a 
complex and time consuming activity that often takes days 
or weeks to complete by a human operator. A test system of 
the present invention can be easily configured to automati- 
cally administer complete SS7 level 2 and level 3 conform- 
ance tests within a matter of hours, without the need for 
human operator intervention. 



[0098] It will be understood that various details of the 
invention may be changed without departing from the scope 
of the invention. Furthermore, the foregoing description is 



for the purpose of illustration only, and not for the purpose 
of limitation — the invention being defined by the claims. 

What is claimed is: 

1. A method for testing a system o^ communications 
network devices, the method comprising: 

(a) reading an abstract command language command from 
a test case, the abstract command language comprising 
a non-device-specific command; 

(b) translating the first abstract command language com- 
mand into a device-specific command compatible with 
an administrative interface associated with a first 
device under test (DUT); and 

(c) communicating the device-specific command to the 
first DUT. 

2. The method of claim 1 comprising receiving a reply 
from the first DUT, and, in response, automatically gener- 
ating a second device-specific command compatible with an 
administrative interface associated with a second DUT. 

3. The method of claim 2 wherein receiving a reply 
message includes receiving a message containing test result 
information. 

4. The method of claim 2 wherein receiving a reply 
message includes receiving a message containing an 
acknowledgment message. 

5. The method of claim 2 wherein receiving a reply 
message includes receiving a message containing status 
information associated with the first device under test. 

6. The method of claim 2 comprising generating a second 
abstract command language command in response to the 
reply message and translating the second abstract command 
language command into the second device-specific com- 
mand. 

7. The method of claim 2 wherein generating a second 
device -specific command includes generating a command 
line interface command compatible with a command line 
interface of the second DUT. 



# check for alarms 
set time 0 

while {$time < 3} { 

eagle cli "REFT- STAT- ALM" result 

if {[iegexp "ALARM. * M AJR" [lindex Sresult (response) 1]]} { 
set eagle_alarm 1 
info line "Eagle alarm verified" 
break 
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8. The method of claim 2 wherein the second DUT is a 
packet network testing device. 

9. The method of claim 1 wherein translating the abstract 
command language command includes accessing a device - 
specific communication interface package that contains 
abstract command language translation instructions. 

10. The method of claim 1 wherein translating the abstract 
command language command into a device-specific com- 
mand includes translating the abstract command language 
command into a command line interface (CLI) command 
compatible with a command-line interface associated with 
the first DUT 

11. The method of claim 1 wherein communicating the 
device-specific command to the first DUT includes trans- 
mitting the device-specific command via a telnet session. 

12. The method of claim 1 wherein the first DUT is a 
packet routing switch. 

13. A method for dynamically testing a system of com- 
munications network devices, the method comprising: 

(a) reading a first abstract command language command 
associated with an operation involving a first device 
under test (DUT); 

(b) translating the first abstract command language com- 
mand into a first command line interface command; 

(c) transmitting the first command line interface command 
to the first DUT; 

(d) receiving a reply message from the first DUT; 

(e) dynamically selecting a second abstract command 
language command based on contents of the reply 
message; and 

(f) executing the second abstract command language 
command. 

14. The method of claim 13 wherein translating the first 
abstract command language command into a first command 
line interface command includes accessing a communication 
interface package associated with the first DUT. 

15. The method of claim 13 wherein transmitting the first 
command line interface command to the first DUT includes 
transmitting the command line interface command using 
remote terminal access software. 

16. The method of claim 13 wherein transmitting the first 
command line interface command to the first DUT includes 
transmitting the command line interface command over a 
wide area network (WAN). 

17. The method of claim 13 wherein receiving a reply 
message includes receiving test result information. 

18. The method of claim 13 wherein receiving a reply 
message includes receiving an acknowledgment message 
associated with a previously-executed command. 

19. A system for testing communications network 
devices, the system comprising: 

(a) a device specific communication interface package 
including information for mapping an abstract com- 
mand language command to a device specific com- 
mand; 

(b) an execution manager for translating abstract com- 
mand language commands to device -specific com- 
mands using the device-specific communication inter- 
face package; and 



(c) a user interface for initiating the execution of an 
abstract-command-language-based test procedure and 
displaying subsequent test results. 

20. The system of claim 19 wherein the device-specific 
command is a command line interface command. 

21. The system of claim 19 wherein the execution man- 
ager is adapted to receive and interpret a response message 
generated by a DUT. 

22. The system of claim 19 wherein the execution man- 
ager is adapted to direct a DUT to be configured with a 
specific operating software version. 

23. The system of claim 19 wherein the execution man- 
ager is adapted to schedule the execution of a test case or a 
test plan. 

24. The system of claim 19 wherein the user interface is 
a graphical user interface (GUI). 

25. The system of claim 19 wherein the user interface is 
adapted to simultaneously display related test plan, test case, 
and test result information. 

26. The system of claim 19 including a version-controlled 
environment for maintaining information associated with a 
test and a device under test (DUT). 

27. The system of claim 26 wherein the version -controlled 
environment is adapted to maintain DUT-specific commu- 
nication interface packages. 

28. The system of claim 26 wherein the version-controlled 
environment is adapted to maintain test plan and test case 
files. 

29. The system of claim 26 wherein the version-controlled 
environment is adapted to maintain test result information. 

30. The system of claim 26 wherein the version-controlled 
environment is adapted to maintain DUT operating soft- 
ware. 

31. The system of claim 19 including a test case editor for 
editing test case information, wherein the test case editor is 
accessible via the user interface. 

32. A system for simultaneously testing a plurality of 
communications network devices, a test tools server for 
maintaining test cases and for storing the system compris- 
ing: 

(a) a test tools server for storing a plurality of device- 
specific communication interface packages, each 
device-specific communication interface package 
including functions for mapping abstract command 
language commands to device-specific command fine 
interface commands; and 

(b) a test management system client for requesting execu- 
tion of test cases on the devices under test; and 

(c) a test controller for receiving the test cases and the 
communication interface packages, connecting with a 
plurality of communications network devices, and 
executing the test cases to test the network communi- 
cations devices. 

33. The system of claim 32 wherein the test tools server 
stores a generic communication interface package contain- 
ing common procedures for accessing the device-specific 
packages and performing device-specific functions during a 
test case. 

34. The system of claim 32 wherein the test tools server 
includes an arbitration engine for dynamically resolving test 
case scheduling conflicts. 

35. The system of claim 33 wherein the generic commu- 
nication interface package includes a startup procedure for 
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accessing the device-specific packages to initialize each 
device involved in a test case. 

36. The system of claim 33 wherein the generic commu- 
nication interface packages includes a cleanup procedure for 
accessing the device-specific packages to free resources on 
each device involved in a test case. 

37. The system of claim 32 wherein the device-specific 
communication interface commands include functions for 



mapping abstract command language commands to tool 
command language commands. 

38. The system of claim 32 wherein the test controller is 
adapted to simultaneously communicate with command line 
interfaces of the communications network devices in order 
to execute test cases and record results. 

***** 
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